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ABSTRACT 

Alternative techniques for analyzing student outcomes 
were investigated. Two traditional statistical methcds (one-way 
analysis of variance and two-way analysis of variance) were compared 
with an innovative. input-output analysis. The input-output analysis 
is a two-step analysis consisting of a regression analysis to 
determine the relationshirfs among input and outfut variables and an 
analysis of variance of the residuals grouped by treatment to 
determine treatment effect. Student outcome data used in the study 
were generated in a typical classroom experiment comparing three 
different methodologies of presenting material to students. Pre- and 
post-measures of studert achievement on a final exam were taken. Some 
measures of student input characteristics, including grade point 
average, age, sex, year in school, and residential status, were also 
taken. Wnile the traditional analysis techniques failed tc show any 
treatment effects, the input-cutput technique showed cne of the 
treatments to be superior to the others. It is suggested that the 
traditional analysis of change scores in experimental research may 
not be directly aprplicable to most research conducted on students. 
The treatment effects on student cutcomes may well be a function of 
input characteristics and an interaction over time for individual 
students, which are not always discernable with traditicnal 
statistical analysis techniques. However, the input-output analysis 
allows for these problems by including other input characteristics in 
the analysis and allowing for the separation of data Fy individuals. 
It is conciuded that student cutccme research cculd benefit from the 
application of the input-output technique. (SC) 
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Student Outcomes 
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This paper has suggested that the traditional analysis of 
change scores in experimental research is not directly applicable 
to most research conducted on students. The treatment effects on 
student outcomes may well be a function of input characteristics 
and an interaction over time for individual students, -and such 
effects are not always discernable with traditional statistical 
analysis techniques. However, an analysis technique known as 
the input-output technique allows for these problems. The re- 
sults of a classroom experiment are presented in three separate 
analyses--one-way analysis of variance, analysis of variance with 
repeated measures, and the input-output technique originally 
applied widely by Astin. It is shown that the traditional 
analysis techniques fail to show any treatment effects. However, 
the input-output technique does show one.of the treatments to be | 
superior to the others. This is explained to be the result of - 
including input characteristics in the analysis, which are not” 
accounted for in the traditional analyses. It is concluded that 
most research having to do with student outcomes would benefit 
from the application of the input-output technique. 
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Introduction: 

Colleges and universities have been confronted with the 
necessity of justifying their programs more and more during re- 
cent years. This author has elsewhere presented the develop- 
ment of this thrust with the restricted funds currently avail- 
able.! rt must be possible to demonstrate that the objectives for 
which a particular program is offered are being achieved. 

When the objectives of the programs of fered by a particular 
college or university are focused on the development of student 
abilities, knowledge, attitudes,. or characteristics, the effect- 
iveness of that program will best be demonstrated by student out- 
comes which show improvement on desired criteria. Student outcomes 
become, therefore, very important in evaluating program effective- 
ness. 

The measurement of student outcomes has often been attempted. 
Many researchers have developed various instruments designed to 
measure certain student characteristics. These measures are * 


often quantitative in nature, providing objective, numerical data 


indicating student performance on the particular criteria being 


assessed. This leads to an approach in program evaluation which 


lelfner, Eliot S., The Use of Student Outcomes in the 
Planning and Resource Allocation Component of the Administrative 
ecision-Making Process in er Education, Unp she 
Dissertation, University of isconaia-uediaca, 1977. 


2For a review of most of the current instruments used for 
measuring student: outcomes, see Chapter Two of Elfner, Eliot S., 
The Use of Student Outcomes., Ibid. 


emphasizes the statistical analysis of measured dvta,” Such 


quantitative information provides administrative decision-makers 
with some systematic data with which to judge program ef fective- 
ness. It should be recognized, however, that the systematic 

data is only one of several inputs to the decision-making pro- 
cess. Program effectiveness should be determined from a combina- 
tion of both qualitative and quantitative inputs. 

The use of quantitative measures of student outcomes in 
program evaluation emphasizes a statistical approach. Most in- 
stances of statistical analyses employ a simple research design, 
coupled with traditional statistical analyses, requiring the 
aggregation of ivfaual change scores in pre- and post- 
measures of res criterion, with an intervening treatment. 
This treatment is usually the program being evaluated. The ith 
pact of the program on the desired student outcome, as determined 
by the criterion measure, is analyzed to determine program 
effectiveness. 

It has been argued elsewhere that such traditional analyses 
of student outcomes results in failure to assess certain very 
pertinent questions about program effectiveness. 4 There is a 
major question regarding the effect of incenine student character- 
istics in interaction with the program treatment. It can often be 

3ror a discussion of the quantitative and qualitative 
methods of program evaluation, See Pace, C. Robert, and Jack 
Friedlander, "Approaches to Evaluation: Models and Perspectives," 
In New Directions for Student Services: Evaluating Program Effect- 
veness, Issue No. l, Spr ng, 1978, PP. 1-18. 
‘see for Lab nadie: Elfner, Eliot S., "Analysis Techniques 
for Assessing the Effect of Higher Education on Student Develop- 


ment Outcomes," paper presented to the Association of Institutional 
Research Nineteenth Annual Forum, May, 1979, San Diego, Ca. 
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asserted that results of the program are a function of the 


incoming students rather than the treatment of the program, and 
that in fact the program has little or no effect on the student 
outcomes being measured. 

Another question raised about the traditional analyses has 
to do with the ceiling and floor and regression effects. 
traditional research designs used in the analysis of student 
outcomes often raise more questions than they answer. It is the 
purpose of this paper to address several methods of analvetiig se 
student outcome data, providing an example. of some real student 
outcome data collected in a traditiona] classroom situation. 

Three separate analysis techniques will be presented. The 
‘first two are traditional statistical methods - one-way ‘and two- 
way analysis of variance.> The other technique to be presented 
is called input-output analysis. 6 student outcome data for this 
paper were generated in a typical gids coomauges tient comparing 
three different methodelegies of presenting material to students. 
Pre- and post- measures of student achievement on a 75 item final 
gent were eaeeny, Also, measures of other student input character- 
istics, such as grade point average, age, sex, year in school, and 
residential status (dorm resident vs. off campus resident) were 
taken. 

Sror a detailed presentation of these two techniques, see 
Hay, William L., Statistics for, the Social Sciences, 2nd ed., 


Holt, Rinehart, and Winston, Inc., (New York, 1973), Chapters 


12 and 13. 


6For a detailed description of the input-output analysis 
technique and its benefits, see Elfner, Eliot S., "Analysis 
Techniques for Assessing ..", Op. cit., pp. 5-7 ; 


The following is a presentation of each of the three 


analysis techniques. The statistical significance of the re- 
sults of each leads to ai ptarant conclusions. A discussion of 
‘the Guelaeawtons of these different conclusions follows. 
Finally, a judgement is’ made about the most appropriate analysis 


technique for use in program evaluation. 


Description of Analyses: 


The experiment which was undertaken for this study consisted 
of three experimental groups. Each member of all three groups. 
was mgasured on several characteristics to establish comparability 
among the three groups. Characteristics measured included academic 
Major, sex, tanidential arrangements, age, and grade point average. 
CHI square and one-way analysis of variance resulted in no. signi- 
ficant Ai teewendey among the groups. | | 

A one-way analysis of variance was conducted on the post-. 


test scores. Figuré One presents the Anova Table. 


SOURCE 


Between Groups 143.75 
Within Groups 7288.19 62.83 
Total 7575069 

XBAR(1) = 48.98 XBAR(2) = 45.46 XBAR(3) = 45.98 


FIGURE ONE 


ONE-WAY ANALYSIS OF VARIANCE TABLE 


The F-ratio associated with the above analysis of variance 
leads one to conclude that there is no significant variance 
among the three groups on the student achievement measure. A 
researcher using this traditional approach to Cer student 
outcomes is likely to infer that the differential treatments 
cause no significant effects on student outcomes. It will be 
shown that this is not necessarily the proper conclusion. 

An analysis of variance and co-variance including repeated 
measures was also conducted. Figure Two presents the results of 


“y 
that analysis. 


SOURCE S UM-OF-SQUARES D.F. MEAN SQUARE F-RATIO 
Grand Mean 386864. 56 386864.56 5320.58 
215.91 : 187.96 1.48 

8434.47 21 
8899.11 - g899.11 | 387.64 
101.55 50.77. ‘2.27 


2663.01 ~ 22:96 


FIGURE TWO 


ANALYSIS OF VARIANCE AND CO-VARIANCE INCLUDING 
REPEATED MEASURES 


j 


Once again, the statistics lead one to conclude that neither 
of the main effects, nor any of the interaction effects are 
significant. Such results would Jmply that the treatments had no 
differential impact on the students' learning. It will be shown - 


that this, too, may not be an accurate conclusion. # 
Ns 


Both of the above analysis techniques require the aggrega- 


tion of data camels from individuals. This aggregation process 
masks the individual changes which result from the experimental 
‘treatments. Individuals change as a result of both treatment 
effects and the interaction of the treatment effects with char- 
acteristics which they themselves bring to the naperinental en- 
vironment. An analysis technique anlek accounts for the effect 
of these input characteristics, as well as the treatment effects, 
would account for these questions. The researcher would then be 
able to more accurately assess the effects of the experimental 
treatment. The Leotneputene technique introduced above provides 
the ability to account for both input and treatment effects. A 
description of the application of this analysis technique to the 
same data as analyzed above follows. 

The initial task in applying the input-output technique is 
to regress the outcome variable on the relevant input data for 
all participants in each of the three experimental groups. In 
this example, the post-test scores were regressed on the pre- 
test scores, and several other potentially relevant input variables. 
Included among these were grade point average, a measure of | 
wvisation to study for the post-test, and age. The results of 
the regression included only the pre-test and grade point average 
as significant input variables. The coefficient of determination 
for the regression was 0.43. 

The next step in this technique is to calculate the residuals 
for each participant. First the expected post-test score is 
calculated using the regressed model, and then subtracted from 


the participant's actual post-test score. These residual scores 
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are identified with each participant, and reflect both their 
knowledge when entering the treatment as well as the effect of 
the treatment. To the Sienk that the residuals of participants 
from one group differ from those of others, effects can be 
attributed to that treatment. By grouping the residuals of 
participants by treatment, simple one-way analysis of variance 
of the residuals can be conducted. To the extent that significant 
results are found, the trea tent can be said to be effective. 

For the data in this study, the residuals here calculated 
for each participant, and grouped by treatment. The analysis of 


variance resulted in a significant F score. Figure Three 


presents the Anova Table for the residuals. 


SOURCE M_O 


5.04 


62.00 
67.04 


0.278 X-BAR(2) =-0.129 X-BAR(3) =-0.168 


The results of this analysis clearly suggest that the first 
treatment group is effected more by the experimental activity 
than either of the other two treatments. Since the residuals 
from all participants seem to be randomly distributed, it is 


clear that the systematic grouping by treatment is significant. 
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Conclusions: 


The results of the above analyses demonstrate that the 


which are not necessarily accurate. This occurs because of 
primary faults with these traditional approaches \ First, 
data are aggregated, leading-to a losa_of information relative 

to individual subjects. nly the aggregated data are manipuiaeed. 


in the statistical analyses. Second, the input characteristics 


of subjects must either be assumed to be randomly distributed, 

or matched. This kind of assumption is appropriate when subjects 
can be randomly het bie or chosen to be matched. However, when 
subjects are self-selected into the Svan tnent groups, as is often 
the case when conducting classroom experiments, and other research 
related to student outcomes, it is not possible to assure that the 
input characteristics of students in the experimental groups are 
matched on all relevant dimensions. Even when choosing subjects 
for matching the several experimental groups, the finite nature 

of the gréup size often precludes accurate matching on all relevant 
input characteristics. Therefore, in order to account for the 
differential input characteristics of subjects in most research 

on student Gytcomes, an analysis technique which minimizes the 
loss of information about subjects seems to be called for. 

The three analyses presented above seem to demonstrate the 
superiority of the input-output analysis technique. Because it 
clearly provides for both the separation of data by individuals, 
and for the input characteristics of subjects, it accounts for a ‘ 
more accurate assessment of the actual Shantee which occur due to 


the experimental treatments. 


{] 


‘The input-output analysis technique does require certain | e 


extra work. By nature, it is a longitutinal assessment tech- 
tates. It requires the collection of data from subjects to be- 
linked to each individual. «In addition, it is a ‘two-step 
analysis -- first a regression cnaivels to determine the rela- 
tionship among input and output variables, and then ‘an ‘analysis | | 
of variance of the residuals grouped by treatment to determine 
treatment effect. These extra demands on the researcher are the 
major disadvantages of the input-output analysis technique: rt 
is suggested that in order to better account for the differences 
in individual characteristics in most research dealings with 
student outcomes, the preferred technique is the input-output 
analysis approach. The above analyses suggest that results of 
traditional analyses may differ from those secured from the input- 
output analysis. 


This paper has suggested that the traditional analysis of 


change scores in experimental research is not directly applicable 
to most research conducted on students. The treatment effects on | 
student outcomes may well be a function of input characteristics 

and an interaction over time for individual students, and such | 
effects are not always discernable with traditional statistical 
analysis techniques. However, an analysis technique known as 

the input-output technique allows for these problems. The results 
of a classroom experiment are presented in three separate analyses 
--one-way analysis of variance, analysis of variance with repeated 


ae 


measures, and the input-output technique originally applied widely 


i2 
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“by, astin.? It is shown that the traditional analysis techniques 
fail to show any treatment effects. However, the input-output 
technique does show one of the Peewee to be superior to the 
others. This is explained to be the result Se including input 
characteristics in the analysis, which are net accounted for in 
the traditional analyses. It is concluded ‘that most research 


having to do with student outcomes would benefit from the 


application of the input-output technique. | 
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